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Abstract 

Background: The TIMI Score for ST-segment elevation myocardial infarction (STEMI) was created and validated 
specifically for this clinical scenario, while the GRACE score is generic to any type of acute coronary syndrome. 

Objective: Between TIMI and GRACE scores, identify the one of better prognostic performance in patients with STEMI. 

Methods: We included 152 individuals consecutively admitted for STEMI. The TIMI and GRACE scores were tested for 
their discriminatory ability (C-statistics) and calibration (Hosmer-Lemeshow) in relation to hospital death. 

Results: The TIMI score showed equal distribution of patients in the ranges of low, intermediate and high risk (39%, 27% 
and 34 %, respectively), as opposed to the GRACE Score that showed predominant distribution at low risk (80%, 13% and 
7%, respectively). Case-fatality was 11%. The C-statistics of the TIMI score was 0.87 (95%CI = 0.76 to 0.98), similar to 
GRACE (0.87, 95% CI = 0.75 to 0.99) - p = 0.71. The TIMI score showed satisfactory calibration represented by %1 = 1.4 
(p = 0.92), well above the calibration of the GRACE score, which showed %2 = 14 (p = 0.08). This calibration is reflected 
in the expected incidence ranges for low, intermediate and high risk, according to the TIMI score (0 %, 4.9 % and 25 %, 
respectively), differently to GRACE (2.4%, 25% and 73%), which featured middle range incidence inappropriately. 

Conclusion: Although the scores show similar discriminatory capacity for hospital death, the TIMI score had better 
calibration than GRACE. These findings need to be validated populations of different risk profiles. (Arq Bras Cardiol. 
2014; 103(2):98-106) 

Keywords: Myocardial Infarction; Propensity Score; Risk Factors; Comparative Study; Acute Coronary Syndrome / 
diagnosis; Prognosis. 



Introduction 

Risk scores correspond to multivariate models used 
in clinical practice to estimate the individual probability 
of unwanted outcomes. Risk estimates have implications 
for clinical management, particularly with regard to 
broad-spectrum diseases, such as acute coronary syndromes 
(ACS). In this context, the main risk scores have been 
developed by the TIMI 1 and GRACE 2 groups. Our group 3 
and others 4 - 5 have previously compared these two scores in 
patients with acute coronary syndromes without ST-segment 
elevation, and it was evident that the GRACE score had better 
prognostic accuracy. However, in patients with ST-segment 
elevation myocardial infraction it has not been established 
which of the two models has better prognostic performance 
with regard to in-hospital death. 
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The GRACE score was derived from a sample composed of a 
combination of patients with ACS without ST-segment elevation 
and patients with ST-segment elevation myocardial infarction. 
The former clinical condition was predominant in the sample 2 . 
Thus, the same score is used in both conditions. On the other 
hand, the TIMI score has distinct models for each of these clinical 
conditions, validated in distinct samples. Therefore, the TIMI 
score for ST-segment elevation myocardial infarction 6 comprises 
variables that are more specific for this clinical scenario. 

To determine which of the two scores has better performance 
in patients with ST-segment elevation myocardial infarction, we 
studied the individuals who were part of the Acute Coronary 
Syndromes Registry (RESCA) and were admitted with this 
condition. In this hospital cohort, with regard to in-hospital 
death prediction, the discriminatory ability and calibration of 
the TIMI score was compared to those of the GRACE score. 

Methods 

Sample Selection 

The patients who participated in this study were part of the 
Acute Coronary Syndromes Registry (Registro de Smdromes 
Coronarianas Agudas - RESCA) of our hospital and provided their 
signed informed consent document. This registry was approved 
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by the Hospital's Research Ethics Committee, according to 
Resolution 466/2012 of the National Health Council. RESCA 
was started in August 2007 and includes patients consecutively 
admitted to the coronary unit of this hospital on the basis of 
the objective diagnostic criteria. The criterion for ST-segment 
elevation myocardial infarction was defined as chest pain 
associated with persistent ST-segment elevation of > 0.1 mv 
(not relieved by nitrates) in two contiguous leads and confirmed 
by elevation of myocardial necrosis markers. In the presence of 
third-degree left bundle branch block, the diagnosis is based 
on the presented symptoms and on necrosis markers. The only 
predefined exclusion criterion in this study was the patient's 
refusal to participate in this registry, which did not occur. 

Study Protocol 

In this study, the outcome predicted by both scores was 
in-hospital death. The patients' data on presentation in the 
emergency room, the electrocardiographic exams performed 
in the first six hours of hospitalization, troponin T or I levels 
measured during the first 12 hours of hospitalization, and the 
first serum creatinine levels were used to calculate the scores. 
The elevation of myocardial necrosis markers, which is one 
of the components of the scores, was defined as troponin 
above the 99th percentile 7 . The Killip classification" was 
applied at time of admission. 

The TIMI score was created and validated in a sample of 
patients with ST-segment elevation myocardial infarction, and 
thus, has specific variables for this clinical condition. In brief, this 
score consists of eight dichotomic variables, with the exception 
of age that adds points in two distinct strata. The presence of 
previous infarction (or left bundle branch block esquerdo), 
major risk factors (hypertension, diabetes, or dyslipidemia), 
weight <67 kg, and reperfusion time >4 hours adds one point 
each. The variables Killip > 1, heart rate >100 bpm, and systolic 
pressure < 100 mmHgadds2 points each. Age adds 2 points if 
between 65 and 74 years and 3 points if >75 years. The final 
score may vary between 0 and 1 4 6 . 

The GRACE score was created and validated in a sample of 
patients with or without ST-segment elevation (1/3 and 2/3 of 
patients, respectively). Therefore, its variables are common to 
both clinical conditions. This score consists of eight variables; 
five of them are analyzed semiquantitatively: different weight 
for each age range, systolic arterial pressure, heart rate, 
serum creatinine, and Killip class; three of them are analyzed 
dichotomically: ST-segment depression, elevation of myocardial 
necrosis markers, and cardiac arrest on admission. The final 
score may vary between 0 and 372 2 . 

Statistical Analysis 

For calculating sample size, we estimated a statistical 
power of 80% (alpha of 5%) for the detection of a difference 
of 0.10 between the ROC curves of both scores, considering 
a correlation between the scores of 0.80. Hence, a sample of 
94 patients was thus calculated 9 . To optimize the statistical power, 
the number of patients with the outcome should be similar to 
the number of patients without the outcome. Because this was 
not expected in our sample, we empirically increased the size 
of the sample by 50%, thus obtaining a total of 1 41 patients. 



The statistical analysis is explained in the diagram of 
Figure 1 . The discriminatory ability of the scores with regard 
to risk of in-hospital death was assessed using ROC curves. 
The area under the ROC curve (C-statistics) represented the 
accuracy of each score to discriminate the survivors from the 
non survivors. C-statistics of the two scores were compared 
pairwise using the Hanley-McNeil test 10 . Next, the optimal 
cutoff-points for each score were determined. Specificity was 
defined as the maximum difference between sensitivity and 
1 . These cutoff-points were used to calculate the prognostic 
sensitivity and specificity (confidence intervals set at 95%). 
The McNemar test was used to compare these parameters 
in the two scores. Score calibration was evaluated by the 
Hosmer-Lemeshow test and by the scatter plot of predicted 
mortality by risk deciles versus observed mortality. 

In a secondary analysis, score performance was assessed 
on the basis of cutoff-points used in the clinical practice, 
which in turn were derived from the studies that developed 
these scores. For this analysis we defined high risk of 
in-hospital death as a likelihood of > 8%, which is above 
the mortality value reported for patients with ST-segment 
elevation myocardial infarction 26 . Low risk of death was 
defined as a likelihood of < 3%, which is below 50% of the 
mortality reported. Based on these definitions, the strata 
that define low, intermediate, and high risk in the TIMI score 
are < 3, 3-4, and > 4, respectively. In the GRACE score, 
the risk strata correspond to < 141, 141-172, and > 172, 
respectively. Initially, the prevalence of patients in each risk 
stratum was described for each score. Subsequently, the 
agreement between the scores in relation to the definition 
of these three risk strata was assessed using the Kappa test 
and the Kendall correlation coefficient. Next, considering the 
definition of high risk as outcome predictor, we described 
sensitivity, specificity, and odds ratios. The Youden index 
was used to evaluate overall accuracy and is calculated as 
(sensitivity + specificity) — 1; a perfect test would have a 
Youden index of 1 . In addition, we used the diagnostic odds 
ratio, defined by the equation (true positive/false negative)/ 
(false positive/true negative). Finally, in an explanatory phase, 
the components of each score were compared between the 
patients who died and those who survived. 

For statistical analysis the SPSS software, version 21 was 
used. The Hanley-McNeil test (comparison of ROC curves) 
was performed using the MedCalc software, version 12. A 
value of p < 0.05 was considered statistically significant in 
all analyses. 



Results 

Sample characterization 

The study included 152 patients, with a mean age of 
63 ±13 years; 72% were men, admitted to hospital within a 
median of 1 71 minutes [interquartile interval (HQ) = 66-402] 
after the onset of symptoms. Of these, 78% were subjected 
to reperfusion in a timely manner, the majority via primary 
coronary intervention. The median door-to-balloon time 
for primary intervention was 132 minutes (HQ = 95-175). 
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Figure 1 - Statistical analysis diagram. 



Other characteristics are described in Table 1, Sixteen 
patients died during hospitalization (lethality of 11%). 
The causes of death were cardiogenic shock in 50% of cases, 
arrhythmic death in 12.5%, renal dysfunction post cardiac 
catheterization in 25% (in patients with hypoperfusion), and 
infection in 12.5%. 

The values of the TIMI and GRACE scores followed normal 
distribution. The mean TIMI score was 3.7 ± 2.3, which 
corresponded to a mortality of 7%' and indicated intermediate 
risk, whereas the mean GRACE score was 116 ± 36, which 
corresponded to a mortality of 1 .6% 2 and suggested low risk. 
We used the cutoff-points determined by the score validation 
studies to define low (< 3%), intermediate (3%-8%), and high 
(> 8%) risk, and we observed that the prevalence of these risk 
strata according to TIMI were 39%, 27%, and 34%, respectively, 
which showed a balanced distribution of patients across risk 
strata. On the other hand, the distribution of patients according 
to the cutoff-points of the GRACE score was asymmetric, with 
80% in the low-risk stratum, 13% in the intermediate-risk 
stratum, and 7% in the high-risk stratum (Figure 2). 

Prognostic Accuracy of Scores 

The analysis of the discriminatory capacity with regard to the 
outcome in-hospital death showed similarity between the TIMI 
and GRACE scores, represented by C-statistics of 0.867 (95% 
CI = 0.76-0.98) and 0.871 (95% CI = 0.75 - 0.99), 
respectively, which indicated good accuracy in both cases 
(p = 0.71 ; Figure 3). In our sample, the optimal cutoff-points 
according to the analysis of the ROC curve were TIMI > 5 
and GRACE >170. Based on these values, the two scores 
exhibited similar sensitivities: 75% (95% CI = 48%-93%) 
for the TIMI score and 81% (95% CI = 54%-96%) for the 



GRACE score (p = 1.0). Similarly, the specificities were 
86% (95% CI = 79%-91%) and 88% (95% CI = 81%-93%), 
respectively (p = 0.82; Table 2). 

The TIMI score exhibited satisfactory calibration, 
represented by % 2 of 1.4 using the Hosmer-Lemeshow test 
(p = 0.92), well better than the calibration of the GRACE 
score, which was % 2 de 14 (p = 0.08). Figure 4 depicts the 
scatter plots of estimated probability versus observed mortality 
by risk deciles. TIMI score data points showed an ascending 
distribution pattern, close to the identity line, which was not 
the case for the GRACE score; this demonstrated that TIMI 
calibration was better than the GRACE calibration. 

Analysis of the Prevalidated Cutoff-Points 

The two scores categorized 59% of patients in the same 
risk stratum (low, intermediate, or high), corresponding to 
a kappa of 0.32 (95% CI = 0.24-0.41), which indicated 
discrete agreement. The Kendall correlation coefficient was 
0.58 (95% CI = 0.49-0.66). 

The analysis of death incidence in the low-, intermediate-, 
and high-risk strata showed an increase in mortality according to 
the TIMI score (0%, 4.9%, and 25%, respectively). With regard 
to the GRACE score, the increase in mortality from the low-risk 
stratum to the intermediate-risk stratum was excessive (2.4%, 
25%, and 73%, respectively), which inadequately characterized 
the predefined stratum as intermediate risk. 

I n a d ichotom ic analysis, the def in ition of h igh risk according to 
the TIMI score exhibited sensitivity of 88% (95% CI = 71%-100%) 
i n predicti ng which patients would present the outcome. This val ue 
was significantly higher than that of the GRACE score (sensitivity 
50%; 95% CI = 26%-75%; p < 0.001). On the other hand, the 
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Table 1 - Sample characteristics 

Variables Values 

Sample Size 152 

Age (years) 63 ±13 

Male sex 110(72%) 

Weight (kg) 76 ±12 

Previous infarction 75 (49%) 

Symptom-to-door time (minutes)* 171 (66^(02) 

Door-to-balloon time (minutes)* 132 (95-175) 

Absent or >4-hour reperfusion therapy 96 (63%) 

Killip class 1.3 ±0.78 

Killip I 124 (82%) 

Killip II 9 (5.9%) 

Killip III 14 (9.2%) 

Killip IV 5 (3.3%) 

Pro-BNP (pg/ml)* 712(274-1817) 

Elevated pro-BNP 67 (44%) 

LV ejection fraction <45% 35 (23%) 

Tri-arterial disease or left coronary artery 70/147 (48%) 

Serum creatinine (mg/dl) 1.1 ± 0.81 

Heart rate (bpm) 81 ±18 

Systolic arterial pressure (mmHg) 150 ± 31 

Previous coronary disease 26 (17%) 

Diabetes 49 (32%) 

Systemic arterial hypertension 1 08 (71 %) 
Reperfusion therapy 

Absence of indication due to late arrival 17 (11%) 

Indication for reperfusion 1 35 (89%) 

Primary angioplasty 1 04 (77%) 

Thrombolysis 15(11%) 

Coronarography without indication for angioplasty 9 (6.7%) 

Absence of strategy due to diagnostic error 7 (5.2%) 
Adjuvant pharmacological therapy 

Aspirin 150 (99%) 

Clopidrogrel or ticagrelor 143 (94%) 

Statin 143 (94%) 

ACE inhibitor 111(73%) 

Beta blocker 99 (65%) 
Risk scores 

GRACE score 116 ±36 

TIMI score 3.7 ±2.3 



'Variables expressed as median and interquartile interval. The remaining 
numerical variables are expressed as mean and standard deviation. Elevated 
pro-BNP: > 450 pg/ml if age < 50 years; > 900 pg/ml if age > 50 years; If 
distribution nao normal, numerical variables expressed as median (interquartile 
interval). LV: Left ventricle; ACE: Angiotensin converting enzyme. 



specificity of TIMI was 72% (95% CI = 65%-80%), moderately 
lower than that of GRACE (specificity 98%; 95% CI = 97%-1 00%; 
p = 0.03). These values of sensitivity and specificity resulted in a 
positive odds ratios and negative odds ratio of 3.1 and 0.1 7 for 
TIMI and 23 and 0.51 for GRACE, respectively. Thus, positive and 
negative TIMI results influence the risk of death, whereas only 
positive results of GRACE have a significant influence. With regard 
to overall accuracy, the Youden indexes of the TIMI and GRACE 
scores were 0.6 and 0.48, respectively. The diagnostic odds ratio 
of TIMI (42) was higher than that of GRACE (1 9). Therefore, the 
accuracy of TIMI using prevalidated cutoff-points was higher than 
that of GRACE. 

Exploratory Analysis of Score Components 

Of the four variables that are common to both scores, three 
exhibited higher values (age, arterial pressure, Killip) in patients 
who died than in those who survived. Heart rate values were 
similar in these two groups of patients. 

Of the four variables exclusive to TIMI, three (previous 
infarction, late reperfusion, and risk factors > 3) were more 
frequent in patients who died than in those who survived. 
Weight (<67 kg) was similar in both groups. Of the four 
exclusive variables of GRACE, only creatinine differed between 
the groups. ST-segment deviation and positive necrosis marker 
are universal characteristics of patients with ST-segment 
elevation myocardial infarction. The fourth variable, death 
after hospital admission, was not observed in any patient. 
These data are shown in Table 3. 

Discussion 

In the present study we compared the performance of 
the two main scores, for predicting death, applied to patients 
with ST-segment elevation myocardial infarction. Our primary 
analysis considered all the ranges of numerical values of the 
scores and demonstrated similar discriminatory ability. However, 
TIMI score calibration was better than that of GRACE. As a 
result, the accuracy of TIMI was better when the cutoff-points 
predefined in the score validation studies were used. 

Our exploratory analysis explains these findings: the 
TIMI score has more variables associated with death. 
This may result from the fact that, unlike GRACE, TIMI was 
derived from a specific sample of patients with ST-segment 
elevation myocardial infraction. The combination of better 
accuracy (although not excessive) and greater simplicity of 
calculation led us to deem TIMI as the preferred score in 
this clinical scenario. 

It is worth revising the significance of the prognostic factors 
under study: discrimination and calibration. Discriminatory 
capacity (C-statistics) describes the score's ability to 
individually distinguish patients who will experience the 
outcome from patients who will not. Calibration, on the 
other hand, relates to the score's ability to predict mortality 
in a given population. It is a collective estimate rather than an 
individual one. Therefore, one might expect the TIMI score, 
which is derived from a sample of patients with ST-segment 
elevation myocardial infarction, to be better calibrated than 
GRACE, which was derived from a sample with only 1/3 of 
patients with this condition. 
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Figure 2 - Distribution of the risk strata according to the TIMI and GRACE scores. 
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Figure 3 - ROC curves for the TIMI and GRACE scores for predicting in-hospital death. Area under the curve for the TIMI score = 0.867 and area under the curve for 
the GRACE score = 0.871 (p = 0.71). 
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Table 2 - Comparison between the TIMI and GRACE scores with regard to discriminatory capacity, calibration, and accuracy according to 
cutoff-points 




TIMI Score 


GRACE Score 


p Value* 


Discrimination 


C-Statistics (95% CI) 


0.867 (0.76-0.98) 


0.871 (0.5-0.99) 


0.71 


Calibration 


X 2 Hosmer-Lemeshow 


1.4 


14 




p Value 


0.92 


0.08 




Optimal Cutoff-Point t 


TIMI >5 


GRACE >142 




Sensitivity (95% CI) 


75% (48%-93%) 


81%(54%-96%) 


1.0 


Specificity (95% CI) 


86% (79%-91%) 


88%(81%-93%) 


0.82 


Positive Odds Ratio 


5.4 


6.5 




Negative Odds Ratio 


0.29 


0.21 




Definition of High Risk (> 8%)t 


TIMI > 4 


GRACE >170 




Sensitivity (95% CI) 


88%(71%-100%) 


50% (26%-75%) 


< 0.001 


Specificity (95% CI) 


72% (65%-80%) 


98% (97%-100%) 


0.03 


Positive Odds Ratio 


3.1 


23 




Negative Odds Ratio 


0.17 


0.51 





t Optimal cutoff-point: best accuracy determined by the ROC curve. 

t Cutoff-point of the validation studies that defined risk of death > 8%, with GRACE score > 170 and TIMI score > 4. 

'Statistical comparison between the two scores using the Hanley-McNeil test for the C-statistics and the McNemar test for sensitivity and specificity. 
CI: Confidence interval. 
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Figure 4 - Correlation between the mortality predicted by the TIMI and GRACE scores and the observed mortality (calibration analysis). 
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Table 3 - Comparison of score components between patients who died and patients who survived 


Variables 


Death 


Survival 


Sample Size 


16 


136 


Common variables 


Age (years) 


77 ±13 


61 ±12 


Systolic Arterial Pressure (mmHg) 


129 ±38 


152 ±29 


Killip 


2.4 ±1.3 


1.2 ±0.59 


Heart Rate (bpm) 


82 ±22 


80 ±17 


Variables Exclusive to TIMI 


Previous Infarction 


5(31%) 


27 (20%) 


Risk Factors > 3 


15(94%) 


106 (78%) 


Reperfusion > 4 hours 


12 (75%) 


84 (62%) 


Weight < 67 kg 


3(19%) 


30 (22%) 


Variables Exclusive to GRACE 


Serum Creatinine (mg/dl) 


1.35 ±0,69 


1.06 ±0,82 


ST segment deviation 


16(100%) 


16(100%) 


Elevation of necrosis marker 


16(100%) 


136 (100%) 


Cardiac arrest 


0 


0 



The similar discriminatory ability observed in our primary 
analysis suggests that the scores are interchangeable with 
regard to distinguishing the patients who will experience the 
outcome from those who will not, provided the cutoff-points 
are adapted to our population. On the other hand, the 
calibration analysis indicated that the performance of 
the TIMI score with regard to estimation of probability of 
outcome is better. By using a basal probability to calculate 
the absolute impact of a clinical decision, this estimate would 
prove useful for clinical decision trees. We acknowledge that, 
in ST-segment elevation myocardial infarction, risk prediction 
does not modify the initial treatment because the indication 
for reperfusion is universal. Still, a well calibrated score has 
practical advantages: low-risk patients subjected to primary 
intervention can be discharged earlier; beta blockers should 
be used carefully used in high-risk patients who tend to 
be hemodynamically unstable; complete revascularization 
via percutaneous intervention may be better than simply 
intervening on the involved vessel in high-risk patients 11 (this 
hypothesis needs to be tested in large randomized studies). 
Second, risk estimation is useful for assistance quality control, 
namely by comparing the predicted mortality in a population 
(score) with the observed mortality in our sample. 

Another aspect that suggested greater usefulness of 
TIMI was the balanced distribution of patients in the low- 
, intermediate-, and high-risk strata, when considering 
the cutoff-points used in the clinical practice. This allows 
applying differentiated treatments to a larger number of 
patients. In contrast, the GRACE score classified almost 
all patients as low risk, which would hinder differentiated 
risk-based treatment. 

Recently, Kozieradzka et al 12 compared these scores in 
ST-segment elevation myocardial infarction and observed 



similar discriminatory ability. However, these authors did not 
analyze score calibration. In addition, that study analyzed 
long-term outcomes (five-year evolution) 12 . We, on the 
other hand, focused our approach on hospital outcomes, for 
two reasons: first, these scores were originally created and 
validated for this type of prediction; second, the prediction of 
hospital outcomes has greater ability to influence acute phase 
treatment. This characteristic and our calibration analysis 
represent the original contribution of our study. 

Next, we discuss the significance of the relatively 
high death incidence (11%) in our sample. As shown in 
Figure 4 (calibration graph), the two deciles of patients who 
experienced higher mortality are the same patients whose 
TIMI and GRACE scores predicted high mortality. On the 
other hand, the graph shows that mortality was low in the 
other eight deciles of predicted low risk. That is, when we 
demonstrated that the calibration was good, mortality was 
high among high-risk patients (according to the scores) and 
was low among low-risk patients. Therefore, it is evident that 
the observed mortality was the result of the characteristics of 
some high-risk subgroups. This is the statistical explanation 
for the observed mortality. In line with this idea, Table 3 
indicates that the severity characteristics of both scores 
were more present in the patients who died. Moreover, it 
should be noted that our sample was limited to patients 
with ST-segment elevation myocardial infarction, a group in 
which mortality was expected to be higher than that in the 
overall GRACE cohort. 

On this subject it should be said that the calibration 
of a prediction model can vary with the incidence of the 
outcome in the population. Therefore, our findings cannot be 
extrapolated with a high degree of certainty to populations 
that are at a significantly different risk for outcome. 
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With regard to the high frequency of late reperfusion, it 
should be noted that this reperfusion time does not refer to 
door-to-balloon time but rather the time elapsed between 
onset of symptoms and reperfusion. Thus, it is expected 
that most patients exhibit a prolonged reperfusion time. 
The original work on the TIMI score (which includes this 
variable) used a sample from a randomized clinical trial, i.e., 
a selected sample, which may explain the favorable rates of 
early reperfusion. 

Ideally, the evaluation of a risk marker should not be 
performed in a context where this marker is used by the 
team who cares for the patient because this leads to the 
underestimation of the observed predictive value. However, 
these scores are known and have been adopted in the clinical 
practice, wh ich makes it i m possi ble to guarantee that they are 
not used in the current registries. Although we acknowledge 
this limitation, we note that it probably occurred with the 
two scores. This lessens this potential bias because the main 
objective was to compare the scores. 

Conclusion 

The discriminatory abilities of the TIMI and GRACE scores 
were similar; however, the calibration of TIMI was better 
than that of GRACE. This result, combined with the greater 
simplicity of the TIMI score, makes this the preferred score 



in patients with ST-segment elevation myocardial infarction. 
In future studies, this conclusion should be validated for 
populations at different risks for the outcome. 
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